Skip to content

gh-162016: Fix communicate() hang when a full stdin pipe is reported writable - #162058

Open
JohnnyWilson16 wants to merge 6 commits into
python:mainfrom
JohnnyWilson16:gh-162016-communicate-nonblocking-stdin
Open

JohnnyWilson16 wants to merge 6 commits into
python:mainfrom
JohnnyWilson16:gh-162016-communicate-nonblocking-stdin

Conversation

@JohnnyWilson16

@JohnnyWilson16 JohnnyWilson16 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

On macOS and FreeBSD, poll() and select() can report a pipe as writable when a write to it would block. communicate() assumed a writable event meant a PIPE_BUF-sized write could not block, so its blocking os.write() to the child's stdin could stall past timeout, or deadlock against a child that was itself blocked writing to stdout.

On macOS this happens once system-wide pipe buffer memory reaches its fixed 16 MiB limit. A new pipe then stays at the size of its first write (512 bytes for communicate()), while readiness is computed as if it could still grow to 16 KiB. It is most likely to be seen on loaded hosts with many pipes in use. The FreeBSD case is a narrower window found by reading its kernel source and has not been reproduced.

Changes

  • Popen._communicate() makes stdin non-blocking while _communicate_io_posix() runs and restores blocking mode afterwards if stdin is still open. A resumed communicate() after TimeoutExpired and direct proc.stdin writes behave as before.
  • _communicate_io_posix() handles BlockingIOError by taking stdin out of the selector for a delay that doubles from 1 ms to 50 ms and resets on the next successful write. A timed retry is needed because the kernel keeps reporting the full pipe as writable, so there is no event to wait for. The other pipes are still serviced during the delay, and activity on them ends it early.

Nothing changes when the kernel reports readiness truthfully. In the affected state, a child that reads stdin but writes no output receives its input more slowly than with a blocking write: about 3 MiB/s in the measurement below.

Tests

Both tests replace _PopenSelector with one that always reports stdin as writable, so they do not need to put the host into the affected state.

  • test_communicate_timeout_stdin_spuriously_writable: the child never reads stdin. communicate() must raise TimeoutExpired on time, must not retry the full pipe in a busy loop, and must leave stdin blocking.
  • test_communicate_stdin_spuriously_writable_last_pipe: the child closes stdout before reading stdin, so stdin is full and is the only pipe left. All input must still arrive.

Verified against the real kernel behavior

On macOS 27.2 (arm64), with the limit reached by one process holding 255 pipes filled to 64 KiB:

Case main This PR
Reproducer from the issue (/bin/cat round trip, timeout=2) hangs returns all 32768 bytes
Child never reads stdin, communicate(timeout=3) TimeoutExpired after 6.0 s, when the child exits TimeoutExpired after 3.0 s, 0.01 s CPU
Child reads 1 MiB of stdin and writes nothing 0.03 s 0.32 s

On some macOS versions, poll() reports a full pipe as writable. The
POSIX communicate() loop then called a blocking os.write() on the
child's stdin, which hung past the timeout and could deadlock against
a child that was blocked writing to stdout.

Set stdin to non-blocking around _communicate_io_posix(), treat
BlockingIOError as "not ready yet", and restore blocking mode if stdin
is still open, so a resumed communicate() and direct proc.stdin writes
work as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@gpshead gpshead left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this attempt at a solution winds up busy-looping wasting CPU under the macOS wide global max pipe buffers limit (see analysis linked from the issue) that triggers the OS bug. if we go this route, we'd need a backoff mechanism as part of the BlockingIOError (EAGAIN / EWOULDBLOCK) exception handler.

Comment thread Lib/subprocess.py Resolved
@bedevere-app

This comment was marked as resolved.

gpshead and others added 5 commits November 15, 2026 11:44
With stdin non-blocking, a pipe that poll() keeps reporting as writable
while it is full made communicate() spin: select() returned at once,
write() raised BlockingIOError, and the loop repeated, using a full core
until the child read its input or the timeout expired.

After a write that would block, take stdin out of the selector for a
delay that doubles from 1 ms to 50 ms and resets on the next successful
write.  The other pipes are still serviced during the delay, and any
activity on them ends it early, since output from the child suggests it
has also consumed input.

Measured on macOS 27.2 with the system-wide pipe buffer limit reached,
for a child that never reads stdin and communicate(timeout=3): 3.00 s of
CPU before this change, 0.01 s after.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Use "is not None" when restoring stdin's blocking mode, matching the
check made before the try block.

Reword the NEWS entry to say that this works around a macOS and FreeBSD
kernel bug that appears under system-wide pipe buffer memory pressure,
most likely on loaded hosts with many pipes in use.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Lead with the affected API and what a user would notice, and drop the
description of how the workaround is implemented.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Explain what stdin_paused_for holds, say why stdin's blocking mode is
restored in terms of what later callers expect, and describe the test's
blocking case without referring to "the fix".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The test selector waited 10 ms in every select() call, which hid a busy
loop.  It now reports stdin as writable without waiting and counts how
often it did so for a pipe that was not, so the timeout test can assert
that communicate() does not keep retrying a full pipe.

Add a test for the case where stdin is full and is the only pipe left:
the child closes stdout before it reads its input, and checks that it
received all of it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@gpshead gpshead added awaiting review needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes and removed awaiting changes labels Nov 15, 2026
@gpshead gpshead changed the title gh-162016: Keep stdin non-blocking in subprocess communicate() gh-162016: Fix subprocess communicate() hanging when a full stdin pipe is reported writable Nov 15, 2026
@gpshead gpshead changed the title gh-162016: Fix subprocess communicate() hanging when a full stdin pipe is reported writable gh-162016: Fix communicate() hang when a full stdin pipe is reported writable Nov 15, 2026
@JohnnyWilson16

JohnnyWilson16 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the analysis @gpshead and for pushing the backoff. You were right about the spin. I pulled your commits and checked them on macOS 26.6 arm64 (pydebug build).

I reproduced the real kernel condition by filling 257 pipes until new pipes stayed at 512 bytes, then ran the issue's cat reproducer. main hangs past timeout=2. This branch returns in 0.01 s with all 32768 bytes.

Same pressure, child that never reads stdin, timeout=2:

version returns after CPU
main 5.01 s (blocked in write() until the child exited) 0.04 s
my first commit 2.00 s 2.06 s (the busy loop)
with your backoff 2.01 s 0.05 s

./python -m test test_subprocess -j0 passes, the two new tests passed 20 of 20 runs, and -R 3:3 shows no leaks.

The backoff looks right to me. stdout and stderr stay registered during the pause, so the child can still drain its output, and the or stdin_backoff loop condition covers stdin being the last pipe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants